Systems and methods for local dimming in multi-modulation displays

ABSTRACT

Dual and multi-modulator projector display systems and techniques are disclosed. In one embodiment, a projector display system comprises a light source; a controller, a first modulator, receiving light from the light source and rendering a halftone image of said the input image; a blurring optical system that blurs said halftone image with a Point Spread 
     Function (PSF); and a second modulator receiving the blurred halftone image and rendering a pulse width modulated image which may be projected to form the desired screen image. Systems and techniques for forming a binary halftone image from input image, correcting for misalignment between the first and second modulators and calibrating the projector system—e.g. over time—for continuous image improvement are also disclosed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 16/910,044 filed on Jun. 23, 2020, which is acontinuations of U.S. patent application Ser. No. 15/995,017 filed onMay 31, 2018, now U.S. Pat. No. 10,694,157 issued on Jun. 23, 2020,which is a continuation of U.S. patent application Ser. No. 15/034,169filed on May 3, 2016, now U.S. Pat. No. 9,992,460 issued on Jun. 5,2018, which is the U.S. National Stage of International Application No.PCT/US2014/062952 filed Oct. 29, 2014, which claims the benefit ofpriority from U.S. Provisional Patent Application No. 61/899,280 filedNov. 3, 2013, all of which are incorporated herein by reference.

TECHNICAL FIELD

The present invention relates to multi-modulation display systems and,particularly, to systems and methods for rendering image and video dataon multi-modulation display systems.

BACKGROUND

Projector systems are now being architected with improvements in dynamicrange and light efficient use. Dual and multi-modulator projectordisplay systems are known in the art. However, additional improvementsare possible in both the rendering and the performance of such displaysystems resulting from improved modeling of the light processing in suchdisplay systems.

SUMMARY

Dual and multi-modulator projector display systems and techniques aredisclosed. In one embodiment, a projector display system comprises alight source; a controller, a first modulator, receiving light from thelight source and rendering a halftone image of the input image; ablurring optical system that blurs said halftone image with a PointSpread Function (PSF); and a second modulator receiving the blurredhalftone image and rendering a pulse width modulated image which may beprojected to form the desired screen image. Systems and techniques forforming a binary halftone image from input image, correcting formisalignment between the first and second modulators and calibrating theprojector system—e.g. over time—for continuous image improvement arealso disclosed

In one embodiment, a projector display system, comprising: a laser lightsource; a controller, said controller receiving input image data andoutputting control signals; a first modulator, said first modulatorreceiving light from said laser light source, said first modulatorreceiving said control signals from said controller such that said firstmodulator is capable of rendering a halftone image of said input image;a blurring optical system; said blurring optical system blurring saidhalftone image received from said first modulator; and a secondmodulator, said second modulator receiving said blurred halftone imagefrom said blurring optical system and receiving said control signalsfrom said controller such that said second modulator is capable ofrendering a pulse width modulated image, said pulse width modulatorimage capable of being projected to form the desired screen image.

In another embodiment, a method for projecting desired screen imagesfrom input image data in a projector display system, said projectordisplay system comprising a light source, a controller, said controllerreceiving input image data and outputting control signals, a premodmodulator receiving control signals from said controller and light fromsaid light source, a blurring optical system receiving light from saidpremod modulator, and a primary modulator receiving control signals fromsaid controller and light from said blurring optical system, the methodcomprising: creating a binary halftone image from said input image data;

creating a blurred binary halftone image from said binary halftoneimage; creating a pulse width modulated image from said blurred binaryhalftone image; and projecting a desired screen image from said pulsewidth modulated image.

In yet another embodiment, a method for calibrating a projector displaysystem, said projector display system comprising a light source, acontroller, said controller receiving input image data and outputtingcontrol signals, a premod modulator receiving control signals from saidcontroller and light from said light source, a blurring optical systemreceiving light from said premod modulator, and a primary modulatorreceiving control signals from said controller and light from saidblurring optical system, the method comprising: receiving input imagedata; computing a halftone image; applying a light field model, saidlight field model based on a PSF model of said blurring optical system;computing a primary image for said primary modulator; displaying ascreen image from said primary modulator; capturing said screen imagewith an image capture device; registering the captured screen image withthe premod grid; comparing said registered captured screen image withsaid input image data; if there are differences greater than a desiredamount, then computing a correction to said PSF model; and applying animproved PSF model for further calibration.

Other features and advantages of the present system are presented belowin the Detailed Description when read in connection with the drawingspresented within this application.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments are illustrated in referenced figures of thedrawings. It is intended that the embodiments and figures disclosedherein are to be considered illustrative rather than restrictive.

FIG. 1 depicts one embodiment of a multi-modulation display system andenvironment in which the systems, methods and techniques of the presentapplication may reside.

FIG. 2 depicts one embodiment of a high level diagram of the opticalprocessing/image processing that may be affected by the dual,multi-modulator display system, as given in FIG. 1 .

FIG. 3A depicts one embodiment of a method for producing a suitablebinary halftone image.

FIGS. 3B, 3C and 3D depict another embodiment of a method for producinga suitable halftone image.

FIG. 4 depicts one embodiment of a technique for generating apulse-width DMD compensation image.

FIG. 5 is one embodiment of a flow diagram of using premod-to-primarymaps, light field models and primary-to-premod maps to produce a primaryregistered light field.

FIG. 6 depicts an array of PSFs that may be captured, modeled, and usedto compute a colored (e.g. red, green or blue) light field.

FIGS. 7A through 7F depict an exemplary rendering of an input imagethrough the various processing modules of the present application.

FIGS. 8 and 9 depict one embodiment of the image rendering process and arefinement process for potential continuous correction/improvement ofthe system.

FIG. 10 depicts an example Point Spread Function (PSF) distribution.

FIGS. 11A-D depict one exemplary image as may be processed by anembodiment of the present application.

FIGS. 12A and 12B depict two charts of image processing based onexemplary input image data.

DETAILED DESCRIPTION

As utilized herein, terms “controller,” “system,” “interface,” and thelike are intended to refer to a computer-related entity, eitherhardware, software (e.g., in execution), and/or firmware. For example, acontroller can be a process running on a processor, a processor, anobject, an executable, a program, and/or a computer. A controller maycomprise a processor and a system memory and the memory may compriseprocessor-readable instructions that, when read by the processor, mayaffect one or more methods and/or techniques disclosed herein. One ormore controllers can reside within a process and a controller can belocalized on one computer/processor and/or distributed between two ormore computers/processor. A controller may also be intended to refer toa communications-related entity, either hardware, software (e.g., inexecution), and/or firmware and may further comprise sufficient wired orwireless hardware to affect communications.

Throughout the following description, specific details are set forth inorder to provide a more thorough understanding to persons skilled in theart. However, well known elements may not have been shown or describedin detail to avoid unnecessarily obscuring the disclosure. Accordingly,the description and drawings are to be regarded in an illustrative,rather than a restrictive, sense.

Introduction

In the field of projector and other display systems, it is desirable toimprove both image rendering performance and system efficiency. Severalembodiments of the present application describe systems, method andtechniques to affect these improvements by employing light fieldmodeling for dual, or multi-modulation display systems. In oneembodiment, light source models are developed and used to advantageouseffect. Camera pictures of displayed images of known input images may beevaluated to improve light models. In some embodiments, an iterativeprocess may accumulate improvements. In some embodiments, thesetechniques may be used on moving images to make live adjustments toimprove image rendering performance.

Dual modulation projector and display systems have been described incommonly-owned patents and patent applications, including:

(1) U.S. Pat. No. 8,125,702 to Ward et al., issued on Feb. 28, 2012 andentitled “SERIAL MODULATION DISPLAY HAVING BINARY LIGHT MODULATIONSTAGE”;

(2) United States Patent Application 20130148037 to Whitehead et al.,published on Jun. 13, 2013 and entitled “PROJECTION DISPLAYS”;

(3) United States Patent Application 20110227900 to Wallener, publishedon Sep. 22, 2011 and entitled “CUSTOM PSFs USING CLUSTERED LIGHTSOURCES”;

(4) United States Patent Application 20130106923 to Shields et al.,published on May 2, 2013 and entitled “SYSTEMS AND METHODS FORACCURATELY REPRESENTING HIGH CONTRAST IMAGERY ON HIGH DYNAMIC RANGEDISPLAY SYSTEMS”;

(5) United States Patent Application 20110279749 to Erinjippurath etal., published on Nov. 17, 2011 and entitled “HIGH DYNAMIC RANGEDISPLAYS USING FILTERLESS LCD(S) FOR INCREASING CONTRAST AND RESOLUTION”and

(6) United States Patent Application 20120133689 to Kwong, published onMay 31, 2012 and entitled “REFLECTORS WITH SPATIALLY VARYINGREFLECTANCE/ABSORPTION GRADIENTS FOR COLOR AND LUMINANCE COMPENSATION”.

-   -   all of which are hereby incorporated by reference in their        entirety.

Exemplary Physical Architecture

In general, a projector with a single Digital Micromirror Device (DMD)may tend to have a limited contrast ratio. To obtain a greater contrastratio, two or more DMDs and/or other reflectors (e.g., MEMS) may bearranged in series. As a DMD may operate as a time-division orpulse-width modulator, operating two or more DMDs and/or reflectors inseries—both acting as pulse-width modulators—tends to require precisetime-division alignment and pixel-to-pixel correspondence oftime-division sequencing. Such alignment and correspondence requirementsmay be difficult in practice. Thus, in many embodiments of the presentapplication, projector and/or display systems may employ differentdual-modulation schemes to affect the desired performance.

As will be discussed in greater detail in the context of an exemplaryprojector display system, a first DMD/reflector—referred to as the“pre-modulator” or “premod modulator”—may spatially modulate a lightsource by means of a halftone image that may be maintained for a desiredperiod of time (e.g., a frame or a portion thereof). This halftone imagemay be blurred to create a spatially-reduced-bandwidth light field thatmay be applied to a second DMD/reflector. The secondDMD/reflector—referred to as the primary modulator—may pulse-widthmodulate the blurred light field. This arrangement may tend to avoidboth requirements mentioned above—e.g., the precise time-divisionalignment and/or the pixel-to-pixel correspondence. In some embodiments,the two or more DMDs/reflectors may be frame-aligned in time, andapproximately spatially frame-aligned. In some embodiments, the blurredlight field from the premod DMD/reflector may substantially overlap theprimary DMD/reflector. In other embodiments, the spatial alignment maybe known and accounted for—e.g., to aid in image rendering performance.

While the present application is presented in the context of a dual,multi-modulation projection system, it should be appreciated that thetechniques and methods of the present application will find applicationin dual, multi-modulation display systems. For example, a dualmodulation display system comprising a backlight, a first modulator(e.g., LCD or the like) and a second modulator (e.g., LCD or the like)may employ suitable blurring optical components and image processingmethods and techniques to affect the performance and efficienciesdiscussed herein in the context of the projection systems.

It should also be appreciated that—even though FIG. 1 depicts atwo-stage or dual modulator display system—the methods and techniques ofthe present application may also find application in 3 or more modulator(multi-modulator) display systems. The scope of the present applicationencompasses these various alternative embodiments.

FIG. 1 is one embodiment of a dual modulating projector display system100 suitable for the purposes of the present application. Display 100may comprise a light source 102—which may comprise one light source(e.g. lamp or the like) or a plurality of point sources of light (e.g.,lasers, LEDs or the like). In the context of a digital movie projector,light source 102 in FIG. 1 may comprise one or more banks of laser lightsources (e.g., 102-1, 102-2, 102-3; 102-1′, 102-2′, 102-3′—where theremay be a plurality of colored light sources that when combined mayrender a white light—e.g., red, green and blue).

Light from source 102 may be piped into optical stage 104—which maycomprise a combiner 104-1 to combine the light from the RGB lasersources and integrating rod 104-2 which may improve the uniformity ofthe light. Light 103 may thereafter be transmitted through a diffuser106 to provide angular diversity to the light. Firstmodulator/pre-modulator 108 may input this light and—under control ofcontroller 120—may provide pre-modulator image processing, as describedfurther herein.

In one embodiment (and as shown in FIG. 1 ), first, pre-modulator 108may be a DMD array that—through a set of optical elements may processseparate color channels (e.g., 108-1, 108-2 and 108-3 for, e.g., red,green and blue channels). For merely exemplary purposes, pre-modulator108 may be a 1.2″, 2K mirror DMD, using standard prism design.Pre-modulator 108 may be controlled to display a binary half-toneimage—e.g., where the pixels are full ON or OFF (where light in the OFFstate may be dumped to offstate light 105). In other embodiments, analogMicro-Electro Mechanical System (MEMS) and/or other analog and/ordigital reflectors may be suitably controlled to redistribute light toform a different type of image.

This half tone image 107 may be transmitted through a Point SpreadFunction (PSF) optic stage 112. PSF optical stage may comprise manydifferent optical elements 110, 114 or the like—e.g., lenses, diffusers,reflectors or the like. It will suffice for the purposes of the presentapplication that PSF optic stage receives the half-tone image from thepre-modulator 108 and provide a desired defocusing of the half-toneimage (109) to the second modulator/prime modulator 116. As with firstmodulator 108, second modulator may be a DMD array that—through a set ofoptical elements may process separate color channels (e.g., 116-1, 116-2and 116-3 for, e.g., red, green and blue channels). For merely anotherexemplary purposes, pre-modulator 108 may be a 1.4″, 4K mirror DMD,using standard prism design.

Prime modulator 116 may receive light 109 and may be controlled bycontroller 120. Controller 120 may employ a light field simulation thatestimates and/or models the combined effect of half-toning and PSF todetermine local brightness on the prime modulator 116 on apixel-by-pixel basis. In other embodiments, such as those employing MEMSreflectors, controller 120 may similarly model the light fieldformation. From this model, controller 120 may calculate, estimate orotherwise determine the pixel values for the prime modulator 116 tomodify the light field to produce the final projected/rendered image.Light 113 may thereafter be transmitted through projections optics 118to form a final projected/rendered image on a projector screen (notshown). OFF light may be dumped to offstate light 111.

In many embodiments, a final image may be produced that is the productof the defocused half-tone image and the prime modulator image. In sucha final image, contrast may be in the range of 15,000,000:1.

One Embodiment of Optical Processing/Image Processing

Having discussed an exemplary projector display system suitable for thepurposes of the present application, it will now be disclosed somemethods and techniques for image processing that may affect theimprovements in image processing and system efficiencies.

In one embodiment, the projector system may create a binary halftoneimage, which may be smoothed by optical components to create a reducedbandwidth version of the desired display image. The shape of the opticalcomponent PSF may determine the properties of the smoothing function.The shape of the PSF may influence display performance and thecomputational requirements of the system. In many embodiments, PSFshaping may have one or more of the following attributes and/or thefollowing guidelines:

-   -   (1) the PSF may smooth the sparsest halftone pattern to a        relatively flat field. This may impose an approximate lower        bound on the size of the PSF;    -   (2) larger PSFs may reduce the spatial frequency at which dual        modulation is active and may result in larger “halos” (as        discussed further herein). This may require larger computational        costs;    -   (3) the PSF may have limited bandwidth and limited rise-times.        Higher bandwidth and rise-times may require greater compensation        accuracy and limit computational approximations;    -   (4) the PSF may be compact and the PSF spatial extent may be        limited. The PSF may decay to zero. A slow decay, or strong PSF        “tails”, may limit image contrast and increase computational        requirements;    -   (5) the PSF may be substantially radially symmetric. Any        asymmetry may be accounted for in the computation.

In one embodiment, the optically blurred PSF may substantially assumethe shape of a Gaussian, or a revolved raised-cosine function, or someother substantially radially symmetric peaked function with limitedspatial extent or the like. FIG. 10 depicts one example (1000) of a PSFdistribution—which may assume a Gaussian-like peak structure 1002 thatmay gradually decay, as seen in tail 1004. In many embodiments, the PSFshould assume limited spatial frequency, limited rise times and/orlimited spatial extent. Spatial frequency and rise times may be usuallycorrelated. Excessive spatial frequency or rise times may require densersampling and greater modeling precision, increasing computationalrequirements. If the PSF varies over the image frame, a set of PSFs maybe used, and a PSF interpolation method may be employed. PSFs with highspatial frequencies that change with PSF position may require a densermodel set for proper interpolation, increasing computationalrequirements and calibration complexity. It may not be desirable to havesharp spikes or ridges on the PSF pulse. Also, it may be desirable thePSF should gradually decay at its perimeter rather than end abruptlythere. A smooth shape will have lower spatial frequencies and longerrise times. The spatial extent of the PSF may determine the size ofcomputation operators. PSFs with broad decaying “tails” may increaseoperator size and therefore computational requirements.

In merely one exemplary embodiment, the PSF represents the blur functionthat is applied to—.g., a 5×5 dither pattern. So, the PSF may be largeenough to produce a relatively flat field from a halftone imagecomprising a 5×5 grid of ones, with all other halftone pixels zero. Ifthe blur function has a substantially Gaussian shape or the like, thenits diameter may range from 10 pixels to 20 pixels. In this example, alower and upper bound may be specified that limits the shape of the PSF.The lower bound may be a raised-cosine pulse and the upper bound may bea Gaussian pulse.

For merely one example, let LB be the lower bound and UB the upperbound. Let “r” be the distance from the center of the PSF, and N thesize of the side of the dither pattern, both in pixels. The pulseamplitude may then be scaled by constants K₁ and K₂ such that the energyfor each pulse is normalized to 1, as follows:

LB(r)=K ₁(½+½ cos(πr/N)) for r<N

LB(r)=0 for r≥N

UB(r)=K ₂ exp(−(r/N){circumflex over ( )}2)

As may be noted, the lower bound decays to zero and the upper bounddecays as a Gaussian. The decay is significant to avoid the accumulationof too much light from PSF tails. It will be appreciated that many otherPSF shapes and functions are possible and that the scope of the presentapplication encompasses all such variations.

Referring attention to FIG. 2 , FIG. 2 depicts one embodiment of a highlevel flowchart 200 for operation of the optical processing/imageprocessing that may be affected with a dual, multi-modulator displaysystem, such as depicted in FIG. 1 . Uniform light 201 may be input intothe display system and a first modulator 202 (e.g., half-tone DMD orother modulator) provides a half-tone image to blurring optics 204.Thereafter, blurred image may be received by second modulator 206 (e.g.,pulse-width DMD or other modulator) further modulates the blurred imageto produce screen image 203. In one embodiment, flowchart 200 tracks aset of processor-readable instructions that may be stored in systemmemory in a controller. A controller may receive image data, produce ahalf-tone image (e.g. at 202), blur the half-tone image (e.g., at 204)and further modulate the image (e.g., at 206) to produce a final image.

In the context of the display system of FIG. 1 , the codewords to eachDMD device may be employed as two variables available to control theimages produced. In one embodiment, the blur function may be performedby an optic system and may be assumed to be constant for all images. Invarious display system designs, the design of the blur function and themethod of halftone imaging and/or encoding may be related and affect theperformance of the display. In one such exemplary system, the followingobjectives/assumptions might be considered in order to determine asuitable choice of halftone imaging and/or encoding and blur functionfollow:

-   -   (1) The contrast ratio of the display may be greater than the        contrast ratio of either DMD alone. However, because of the        halftoning and blur function, the full contrast of the display        may not be achieved for high-spatial frequency components of an        image. Consequently, some bright-clipping or dark-clipping might        exist in displayed images. Visual degradation due to light field        modeling errors may desirably be minimized.    -   (2) No bright clipping: In one embodiment, the blurred light        field incident on the primary modulator/DMD may be everywhere        greater than the input image, the desired screen image. The        primary modulator/DMD may attenuate the light field. In some        embodiments, some image pixels that are greater than the blurred        light field may be bright-clipped. While no bright clipping may        be a goal, the blurred halftone image may be substantially        brighter than the desired screen image nearly everywhere, and        there may be portions of the blurred halftone image that are        less bright than the desired screen image and bright clipping is        possible.    -   (3) Limited dark clipping: In one embodiment, the blurred light        field may be attenuated by the primary DMD to produce the        desired screen image. Because the primary DMD may have a limited        contrast ratio, the system may only produce image pixels greater        than the blurred light field divided by the primary DMD contrast        ratio. In some embodiments, image pixels that are less than this        may be dark-clipped.    -   (4) Small halo: Halos are dark clipping around a bright object        on a dark background. Given that a small bright object on a        black background is not bright clipped, the blurred light field        at the bright object may be greater than the bright object.        Because the light field may have reduced spatial bandwidth, it        may not be dark very near the bright object. The level of the        light field close to the bright object may be reduced by the        primary DMD as much as possible, but might still be greater than        the desired screen level, causing dark clipping. In some cases,        the dark clipping may represent elevated levels above true        black, often exhibiting loss of dark detail and loss of        contrast. Halos are the primary visual artifact caused by dark        clipping, but dark clipping may occur in any local region where        the blurred light field cannot represent a high-contrast, high        frequency pattern. The spatial extent of a local region may be        determined by the bandwidth of the light field which may be        determined by the size of the blur kernel or PSF.    -   (5) Adequate local contrast: The bandwidth of the light field        may be determined by the size of the blur PSF. A smaller PSF        allows for a higher bandwidth light field. But a smaller PSF        must be paired with a denser halftone pattern. A denser halftone        pattern may be associated with a smaller dither pattern size; it        may have fewer discrete levels and a higher first non-zero        level.    -   (6) The above objectives may compete. As such, many variations        and/or embodiments may be possible and/or desirable. This will        be discussed further herein with respect to DMD contrast ratios,        PSF size and local contrast considerations.

Halftone Imaging and Light Field Modeling Embodiments

On a real display, the blurred light field frame created by the premodDMD may not be aligned with primary DMD frame. The light field image maybe slightly rotated, shifted, or scaled to provide overscan at frameedges. It may also be warped due to the blur optic and other optics. Forsuch possibilities, a premod-to-primary map that maps points on thepremod DMD to points on the primary DMD may be measured and applied as amapping—e.g., as a Look-Up Table (LUT) or the like. In a real display,the premod-to-primary alignment may drift due to the average premodimage level. The alignment may change slowly in response to a premodimage level change and may not be easily predicted. The primary DMDshould desirably compensate the blurred light field to produce thescreen image by using a model of the light field. The accuracy of thelight field model may depend much on the accuracy of thepremod-to-primary map used to transfer the blurred light field modelfrom the premod pixel grid to the primary pixel grid for compensation.An inaccurate map, however, may result in strong artifacts in the screenimage. The premod-to-primary alignment drift is typically an undesirableproperty of the display that greatly influences the algorithm used tocreate the halftone image.

In one embodiment, the control algorithm used to form the screen imagemay choose premod and primary DMD codewords to form such desired screenimages. In this embodiment, choosing premod DMD codewords is akin tochoosing the blurred light field, and mostly determines the primary DMDcodewords since these should desirably compensate the light field toproduce the desired screen image. The blurred light field may beeverywhere greater than the input image to avoid bright-clipping,and—when divided by the primary DMD contrast ratio—should be everywhereless than the input image to avoid dark-clipping. Equivalently, theblurred light field should be everywhere greater than a bandlimitedupper bound of the input image and less than a bandlimited lower boundtimes the contrast ratio of the primary DMD. The band-limit requirementis imposed by the halftone encoding and blur function.

In some image regions, the characteristics of the image may be such thatthe upper bound and lower bound conditions may be satisfied. In otherimage regions, they may not both be satisfied. In both instances, achoice of light field is presented. When both conditions can besatisfied, there usually exists a range of light fields to choose from.When both conditions cannot be satisfied, a light field may be chosenthat violates at least one of the conditions. In one embodiment,violating the lower bound may be usually preferred since bright-clippingtends to be more visually apparent than dark-clipping, but depending onimage characteristics, either or both conditions may be violated.

In some embodiments, PSF shape change and premod-to-primary alignmentdrift may both greatly influence the choice of light field. For aparticular image region, there might exist in the set of light fieldsthat satisfy the upper and lower bound conditions a light field that issubstantially constant over the region. There may also exist lightfields that vary spatially, and some more than others. A light fieldthat varies may be susceptible to modeling errors caused by apremod-to-primary map that does not accurately represent the alignmentposition or a PSF model that may not accurately represent the PSF shape.A flat light field is relatively unaffected by these effects.

In one embodiment, the algorithm used to choose the light field may bedesigned to achieve two competing goals. One goal may be to avoid imageerrors due to bright and dark-clipping; the other goal may be to avoidimage errors due to miscompensation by light field modeling errors. Whenlight fields that do not violate the upper and lower bound conditionsexist, the algorithm may choose the one that varies least. But it couldchoose a light field that does violate the conditions but varies muchless than that one. This choice might be better because the resultingbright or dark-clipping is visually preferable to the image errors thatwould be present by inaccurate modeling of the light field that variesmore.

To determine the light field, the algorithm may proceed according to thefollowing rules:

-   -   (1) For any image region, set the light field to the maximum        level unless the image region may be excessively visually        degraded due to dark-clipping. Such degradation may be        determined by observation, heuristically or any other suitable        manner.    -   (2) For any image region that requires a level other than the        maximum level, set the level as near the maximum level as        possible and minimize the variation of the light field as much        as possible without excessively degrading the region due to        bright or dark-clipping.

These rules may be employed and/or modified according to the followingreasons and/or conditions:

-   -   (1) The image regions that have the light field set to a        constant level are not susceptible to compensation errors caused        by premod-to-primary misalignment or PSF shape change.    -   (2) The display may be calibrated at the maximum level. When the        light field is held at this level, the premod-to-primary map may        accurately represent the alignment and the PSF model may        accurately represent the PSF shape. Keeping the light field at        this level as much as possible reduces the deviation of the        alignment and PSF model. In image regions where the light field        is not constant, compensation errors due to light field model        errors may be limited. These errors tend to degrade the image        excessively.    -   (3) Many image regions do not have very high contrast and may be        achieved at the maximum level with insignificant dark-clipping.    -   (4) Compensation errors tend to be less visible in darker        regions. When the light field cannot be set to the maximum        level, the image region is probably very dark.

Halftone DMD Encoding Embodiments

In one embodiment, the pre-modulator/halftone DMD may spatially modulatethe uniform light field to produce a halftone image;—e.g., in which allpixels are either ON or OFF for the entire frame time or a portionthereof. The resulting halftone image—appropriately blurred—may producesufficient light levels on the primary modulator/pulse-width DMD,especially if bright-clipping is desired to be avoided. Since thepulse-width DMD may only reduce light levels, the blurred halftone imageshould be substantially everywhere greater than the desired screenimage—e.g., the input image. In some circumstances when an image featuresuch as a very bright dot on a black background may force theunavoidable condition of choosing either bright or dark-clipping, somebright-clipping may be intentionally allowed and the blurred halftoneimage would not be greater than the input, particularly the dot.

In one embodiment, to achieve low light levels and to avoid halos, theblurred halftone image may be set to be slightly greater than thedesired screen image. So, the blurred halftone image may substantiallybe a bandlimited minimum upper-bound on the desired screen image—e.g.,with the bandwidth limited by the optical blur. One embodiment (asfollows) tends to produce a bandlimited upper-bound on an image. It maynot be a minimum upper-bound, but may have a similar performance. Thisrelaxation may be desirable as a true minimum may be harder to achieve,although possible. In this embodiment, it may suffice that the “nobright clipping” property is substantially preserved.

In this embodiment, the halftone image may be formed by using a spatialdither pattern. The dither pattern may be defined over a rectangularblock of pixels and may be repeated over the entire image frame bytiling the pattern. The size of the pattern may be related to the sizeof the blur kernel, since the kernel smoothes the pattern. The size ofthe kernel may also determine the minimum non-zero light level—e.g., onepixel of the dither pattern turned ON and all others OFF may produce theminimum level.

Table 1 below shows one exemplary 10×10 pattern showing the levelindexes. For a given level index, the numbered pixel and all lessernumbered pixels are turned ON while all greater numbered pixels are OFF.When a given level pattern is blurred, the result tends not to be flatand the modulated field may have some minimum. Table 2 below shows thenormalized minimum light levels for the each level index of Table1—showing the light level for the previous index. It will be appreciatedthat other pattern sizes and other spatial dithering patterns arepossible and are encompassed by the present application.

TABLE 1 Spatial Dither Pattern Exemplary 1 93 17 69 33 4 96 20 72 36 6137 77 29 89 64 40 80 32 92 13 85 5 45 53 16 88 8 48 56 73 25 65 97 21 7628 68 100 24 49 41 57 9 81 52 44 60 12 84 3 95 19 71 35 2 94 18 70 34 6339 79 31 91 62 38 78 30 90 15 87 7 47 55 14 86 6 46 54 75 27 67 99 23 7426 66 98 22 51 43 59 11 83 50 42 58 10 82

TABLE 2 Normalized Minimum Light Levels for Table 1 Pattern 0 0.9199140.159947 0.679884 0.319955 0.029713 0.949726 0.189717 0.709671 0.3496710.599925 0.35989 0.759885 0.279915 0.879896 0.62969 0.389651 0.7896650.309641 0.909666 0.119936 0.839911 0.039901 0.439936 0.519923 0.1496970.869623 0.069613 0.469654 0.549684 0.719902 0.239875 0.639914 0.9599190.199903 0.749686 0.269715 0.669626 0.989631 0.229615 0.479913 0.3999080.559908 0.079919 0.799928 0.509626 0.429714 0.589671 0.109678 0.8296870.019936 0.939929 0.179932 0.699874 0.339925 0.009668 0.929686 0.1696730.689643 0.329627 0.619936 0.379903 0.7799 0.299927 0.899886 0.6096450.369611 0.769624 0.289596 0.889622 0.139927 0.859917 0.05989 0.4599250.539903 0.129655 0.849579 0.049569 0.449609 0.529644 0.73988 0.259890.65992 0.979956 0.21991 0.729642 0.249675 0.649582 0.969587 0.2095710.499944 0.419922 0.57992 0.099932 0.819934 0.489581 0.409674 0.5696270.089634 0.809642

In this embodiment, for any particular input pixel, the level of thecorresponding pixel of the blurred halftone image should be greater. Toachieve the desired greater level at that pixel, all nearby pixels ofthe input image within the spatial extent of the blur kernel may beevaluated—e.g., any of those nearby pixels with a level less than thedesired level may be turned ON. One embodiment of this method may beaffected as follows:

-   -   (1) For any particular input pixel, choose a level index such        that the level of the full-frame light field is greater than the        pixel level. For example, it is possible to choose a level        index—e.g., for the entire frame—that creates a halftone pattern        that, when blurred, exceeds the pixel level.    -   (2) Given this full-frame halftone pattern, all pixels whose        PSFs do not contribute light to the particular pixel may be        turned OFF without affecting the level at the particular pixel.

It should be noted that this method may not produce halftone tiles withparticular levels, giving the halftone image a blocky appearance.Rather, individual pixels may be turned ON or OFF, depending on theirindex and proximity to image features. In other embodiments, it may bepossible to affect error diffusion and/or local blue noise—e.g., wherethe halftone grid may be locally thresholded by the corresponding pixel.

It should be appreciated that while one embodiment may be affected by anordered dithering, it may be linked with a dilation to achieve anupper-bound. Smoothness may be a concern at the lowest levels—e.g., suchas only one pixel on for the dither pattern. It may be possible to applyother approaches, such as blue noise and/or FM-dithering, for differentsmoothness effects. For another example, consider a small bright objectat less than full brightness on black background. In this case, the halointroduced may be wider than desired. The dilation area may not be fullypopulated with ones. A more compact area with all ones may show less ofa halo because halo width of the display is greater than eye glarewidth. Reducing the brightness of small bright objects may reduce halowidth, rather than just reduce halo brightness.

FIG. 3A shows one embodiment of a method for producing a suitable binaryhalftone image. Halftone image module 300 may receive input image data301 and may dilate the image data to the extent of the blur kernel at302—to produce x(m,n), the dilated input image. The resulting binaryhalf-tone image, b(m,n), may be set to b(m,n)=1, ifx(m,n)>htLevel(m,n)—where htLevel(m,n) may be given as the values inTable 2 as the map—e.g., tiled over the entire image frame. The binaryhalf-tone image may be returned as b(m,n). The embodiment as depicted inFIG. 3A may be employed if the light field under specialized and/orsimplified conditions—e.g., when the light field is turned fully ON.

In one embodiment, the dilation operator may be employed to achieveclose to the min upper bound. Other embodiments may employ nonlinearfilters that may provide a max of elements under the kernel.

Embodiments Employing Upper and Lower Bounds

In situations in which the light field may be other than fully ON (e.g.,the light is some greyscale value less than fully ON, for reasonsdiscussed herein), then it may be desirable to employ upper and lowerbounds.

FIG. 3B depicts one method for determining an upper bound of a binaryhalftone image. Input image 301 may be input into block 302 that dilatesto blur kernel extent. That intermediate result may be subject to thethreshold in 304 b—i.e., b(m,n) may be set to x(m,n), if greater thanhTLevel(m,n). This block would set the upper bound binary halftone imageat 303 b.

In one embodiment, if the upper bound binary halftone image is found bythis method, then it is likely that no input image pixel will bebright-clipped. However, some pixels may be dark-clipped.

FIG. 3C depicts one method for determining a lower bound of a binaryhalftone image. The lower bound binary halftone image is found similarlyto the upper bound image. The same dither pattern and threshold map maybe used with an offset of one level added to each threshold. Rather thandilating the input image, it may be eroded. Erosion is a process similarto dilation, except that erosion may occur around the dark pixels. Forexample, if the system inverted the image and then performed dilationnormally and then re-inverted the resultant image, the result would bean erosion of the image. For dilation, in some region near a pixeldefined by some shape such as a disk, it is possible to set the otherpixels to this pixel's level if it is greater. For erosion, it ispossible to do the same, except set the other pixels to this pixel'slevel if it is less. Dilation tends to set a pixel to the maximum in alocal region;

erosion tends to set a pixel to the minimum in a local region. Afterthat, rather than turning halftone pixels on when corresponding dilatedimage pixels are greater than thresholds, they may be turned off if lessthan thresholds.

In FIG. 3C, the input image 301 is multiplied (at 306) by the contrastratio of the primary DMD 305 to establish the light field level at whichthe primary DMD may achieve the lower bound. The image may be eroded in302 c and may be thresholded as depicted in 304 c. This scaling of theinput image greatly increases the image levels and very often in commonimages makes the levels greater than the maximum light field level. Thelight field can be set to the maximum level everywhere this is true.

If the lower bound binary halftone image is found by this method, noinput image pixels will likely be dark-clipped, but some may bebright-clipped.

Combining Upper and Lower Bounds for Form the Halftone Image and LightField

The upper and lower bound binary halftone images may be combined to formthe halftone image and light field according to rules described herein.As depicted in FIG. 3D, the upper bound image 303 b and lower boundimage 303 c may be pointwise OR'ed (at 302 d) such that the binaryhalftone image (303 d) may be the greater of the upper bound and lowerbound images.

If the upper and lower bound binary halftone image are combined by thismethod, no input image pixels will likely be bright-clipped, but somemay be dark-clipped. The blurred light field level may be set to thegreatest level permitted by the upper and lower bound methods such thatno bright-clipping exists but dark-clipping may exist due to the limitedspatial bandwidth of the light field.

Alternative Upper Bound/Lower Bound Method Embodiments

For alternative embodiments, several enhancements may be applied thatmay improve image quality. For example, in the method described above,there may be created a strict upper bound light field image. However, insome image regions, limited bright-clipping might be preferred inexchange for less dark-clipping for one possible upper boundenhancement. An example image feature is a small bright dot on a verydark background such that a visible halo is present.

For some lower bound enhancements, it should be noted that the methoddescribed above may create a strict lower bound light field image.However, in most images, a strict lower bound is not necessary. Due toveiling eye glare, an observer will perceive many very low-level pixelsto be at a higher level. Often, raising some low level pixels can beperceived, but does not appreciably degrade the image, especially darkpixels in bright areas.

In an image region, the level of the darkest pixels directly determinesthe amount of image area that can set to the maximum light field level.Displaying a pixel at less than the level attainable by the primary DMDalone requires lowering the light field. Some amount of dark-clippingmay be preferred instead of image errors due to premod-to-primaryalignment or PSF shape change. Exchanging some dark-clipping for moremaximum light field area is often a favored trade-off.

One possible enhancement is to use a veiling glare function to determinehow high the lower bound can be raised such that dark-clipping cannot beperceived. This method tends to raise the light field such that it is atmaximum for more image area.

Another possible enhancement is to use a “safe contrast ratio” ratherthan a strict lower bound. Compared to the strict lower bound, thismethod tends to raise the light field such that it is at maximum formore image area. Rather than use the lower bound times the primary DMDcontrast ratio to determine the maximum the light field may be, acontrast ratio for a local region is chosen to be sufficient, or “safe”,for the purpose of allowing dark-clipping with the assumption that itwill not be perceived visually. Instead of the lower bound times theprimary CR, an average of a local region times the “safe contrast ratio”is the maximum the light field may be. By using a lower “safe contrastratio”—rather than using the primary DMD contrast ratio, there may tendto reduce the image errors from premod to primary alignment or PSF shapechange, possibly at the expense of high spatial frequency dark-clipping.In addition, using a smaller erosion operator may be useful in reducingthe image errors due to premod to primary alignment or PSF shape change.

Light Field Model and Pulse-width DMD Embodiments

The primary modulator/pulse-width DMD modulates the blurred halftoneimage light field to produce the desired screen image. The Pulse-widthDMD can only attenuate light—so the light field may be an upper bound onthe desired screen image to prevent bright clipping. In addition, toprevent dark clipping, the light field may be a minimum upper bound. Theblurred halftone image light field may be computed, estimated orotherwise modeled using a model of the optical process. In oneembodiment, the optical process may be assumed to be only the blur—e.g.,the pre-modulator-to-primary modulator alignment may be ignored. In someembodiments, this may be the overall registration error.

FIG. 4 depicts one embodiment of a technique for generating apulse-width DMD compensation image. Binary half-tone image (303, e.g.,from FIG. 3 ) may be input into a blur model 402 and may be reciprocatedat 404. The pulse-width DMD compensation image may be determined bydividing the input image by the modeled blurred halftone image lightfield—e.g. multiplying (at 406) the input image 301 by the reciprocal ofthe blurred half-tone image light field.

Embodiments for Accommodating Varying PSF Shapes

On the real display, the PSF shape for a given premod pixel may dependon its position on the premod frame. The blur optic may not blur allpremod positions the same. The PSF for pixels in a local area may beassumed to vary little and all pixels may be assumed to have the sameenergy (e.g., given a uniform light field incident on the premod).However, on a real display, each PSF may tend to be different. In oneembodiment, for a 2K frame, each PSF may be modeled separately, and/orapplied to a local portion of the image area—e.g., resulting in 2million PSFs that might be captured, stored, modeled, and usedcomputationally. Other embodiments may provide a simplifying model toreduce this complexity. Because the PSFs in a local region tend to besimilar, a single PSF model is used to represent all PSFs—e.g., at leastin local areas and/or local portions of the image area. Such,potentially localized, PSF models may be measured or otherwise modeledto provide suitable PSF models.

Light Field Model Embodiments

The primary DMD compensates the blurred light field to produce a finalscreen image. In one embodiment, light field compensation may beperformed on the primary DMD pixel grid. For this compensation process,the blurred light field may be represented on the primary pixel grid.However, the light field is formed by blurring the halftone image whichis on the premod pixel grid. In addition, the premod and primarymodulators may not be aligned.

To affect a suitable compensation process, there are two possiblealternative embodiments from which to choose. A first embodiment mightbe to model the light field on the premod grid and then map it to theprimary grid. A second embodiment might be to model the light field onthe primary grid by modeling the PSFs associated with each premod pixelon the primary grid. While the present application encompasses bothalternative embodiments, it will now be described the firstembodiment—i.e., to modeling the light field on the premod grid and mapit to the primary grid. In one embodiment, it may be possible to mappoints on the primary grid accounting for geometric and/or opticdistortions.

This first embodiment may be selected for the following reasons:

-   -   (1) because the PSF may substantially maintain its shape in a        local area. Thus, the light field may be modeled in the local        area on the premod by a standard convolution process on the        halftone image using a single PSF for the entire area.    -   (2) because of the premod-to-primary misalignment, the PSFs in a        local area on the primary may have different sample phases and        these may need to be accounted for. In some embodiments, because        the premod and primary may not be aligned, there may be some        sample-phase shift moving the inherently premod-grid-aligned PSF        models to the primary grid.    -   (3) If modeled on the primary, even though the PSF shape does        not change in a local area, different PSFs may need to be used        when computing the convolution because of the sample phase        change.    -   (4) The PSFs are inherently premod referenced. More PSFs would        need to be modeled and recorded for the primary than for the        premod.    -   (5) Modeling the light field may tend to be computationally        costly. For a practical implementation, the PSFs may need to be        subsampled and approximated. This might be simpler if performed        on the premod grid.    -   (6) Mapping the modeled light field from the premod to the        primary has a computational cost, but it may be less than the        cost of modeling the light field on the primary.    -   (7) Modeling the light field on the premod may be affected by        mapping the modeled light field from the premod to the primary.        It may be desirable that this map be accurate.

For a particular display, the premod-to-primary alignment is fixed. Ifthe map has errors, they may be fixed. For example, the errors may beaccounted for by modifications to the PSFs during the calibrationprocess—e.g., an offset error in the map may be countered by an offsetin the PSF model at that frame position.

In addition, because of premod-to-primary misalignment, the input imagemay be mapped to the premod grid for the process of computing thehalftone image. This process may not require as much accuracy as mappingthe light field to the primary. FIG. 5 depicts the afore-mentioned firstembodiment. The system—under controller direction—may receive inputimage 301. A primary-to-premod mapping 502 may be applied prior tocomputing the half tone image at 504. A light field model may be appliedat 506 and then a premod-to-primary map at 508 may be applied to producea primary registered light field 503. In some embodiments, the primarymay be one resolution (e.g., 4K) and the premod may be anotherresolution (e.g., 2K); but processing may be able to affect othermappings. For example, the processing may affect 2K/2K mapping—but the2K primary may be upconverted to 4K by the projector system. Othermappings are, of course, possible.

Frame Partitioning for Light Field Model Computation

As mentioned above, since the PSF may be substantially unchanged withina local area, the same PSF model may be used in a local area to computethe light field. Thus, the image frame may be partitioned intorectangular regions and each region may be processed using a

PSF that best represents the PSFs in that region. After each region ismodeled, the regions may be joined together to form the full-frame lightfield. If the seams are visible, either more regions may be used—oroverlapping regions may be computed and then blended together. FIG. 6depicts an array of PSFs that may be captured, modeled, and used tocompute a colored (e.g. red, green or blue) light field. The frame maybe partitioned so that the PSFs are at the center of each region.

Temporal Repositioning of Halftone Pattern

In one embodiment, a halftone dither pattern may be tiled over the imageframe. The choice of position or phase of the pattern on the frame mayvary accordingly. Since the screen image may have errors due to lightfield modeling or other causes, the dither pattern may be repositionedeach frame—e.g., so that errors may be disguised. For example, thedither pattern may be repositioned each frame according to the patternindexes. As errors may be periodic with the dither pattern tile size,moving the pattern around according to the indexes may be affect hidingerrors satisfactorily. The dither pattern may also be repositioned todisguise the light field modeling errors caused by the sparsest halftonebit patterns; these bit patterns might cause the most visible errorsbecause they, when blurred, tend to produce light fields that vary morethan denser bit patterns.

Overview of PSF Adjustment and Display Calibration

In practice, a real display may need to have its PSF shape adjusted andmay need to be calibrated. In one embodiment, PSF adjustment may beperformed before the calibration—e.g., by manipulating opticalcomponents. After a suitable PSF adjustment, the calibration process mayoccur for each color channel to measure and record the premod-to-primarymap and the PSF model set. For calibration, a camera may be used to takepictures of the screen while calibration images are displayed.

In one embodiment, to measure the premod-to-primary map, pictures of twodifferent calibration images may be taken. The first calibration imagemay be a PSF array image that is formed by displaying a dot array on thepremod with the primary fully open. The premod pixel positions of thedots are known. This combination shows a screen image of an array ofPSFs, as depicted in FIG. 6 . The second calibration image may be a dotarray image that is formed by displaying a dot array on the primary andall ones on the premod. The primary pixel positions of the dots areknown. This combination shows a screen image of an array of dots.

Pictures of these two images may be taken without moving the camera.Since the premod and primary pixel positions of the dots are known, acamera-to-premod map and a camera-to-primary map are found. These twomaps may then be composited to form the premod-to-primary map for thecolor channel. If the camera is not moved between pictures taken foreach color channel, the camera-to-primary maps for the different colorchannels may be used to align the primary DMDs with each other, aligningthe color channels with each other.

To find an initial PSF model set, the first calibration image may beused, as depicted in FIG. 6 . The camera-to-premod map may be used tomap the PSFs to the premod grid, where they are recorded to form the PSFmodel set. The models are referenced to the premod grid because that iswhere they are used to construct a model of the blurred light field. Forthe purpose of constructing the blurred light field, each PSF in the setrepresents all the PSFs in its local processing region. Acquiring modelsof PSFs this way can easily result in models with significant errors.

During the calibration process, the premod-to-primary alignment and PSFshape should preferably be held constant. Because the algorithm attemptsto keep the premod image at maximum level (white), a white premod imageis used to hold the alignment and shape change constant. Before thecalibration process begins, the display is conditioned with the whitepremod image. The dot test patterns may not have white premods, so thedot patterns are displayed only long enough to take pictures and thenthe white premod image is reapplied to maintain constant conditions.

Additional Display Calibration Embodiments

As mentioned, display calibration may include the process of creating aPSF model set. The initial PSFs may be acquired by taking pictures ofindividual PSFs displayed on the screen.

However, acquiring models of PSFs in this way may result in models withsignificant errors. Since the PSF models may be used to model the reallight field incident on the primary DMD, it may be desired that theyaccurately represent the real PSFs. In some dual DMD modulator projectorsystems, these systems may have relatively small PSFs with steep slopes,which may need accurate models. In one embodiment, it may be possible tocreate a PSF model refinement as a process of improving existing PSFmodels by evaluating the errors produced when the models are used todisplay screen images.

In one embodiment, a single PSF may be captured by camera. This PSFimage may be used as the preliminary model of the PSF for the halftonepixel that produced it. As it may not be a perfect representation of thereal PSF, when it is used to model the light field produced by thehalftone image, the light field may have errors.

For a particular image, the light field may be formed by thesuperposition of all the PSFs produced by the halftone pixels that areON. Such PSF model errors may produce light field errors depending onthe distribution of ON-pixels in the halftone image. Some images may bemore susceptible to PSF errors than others. Generally, images thatproduce a relatively flat light field may be less susceptible to PSFmodel errors, and images that produce a varying light field may be moresusceptible.

FIGS. 7A to 7F are exemplary images to illustrate PSF modeling andrefinements. FIG. 7A depicts a single area illuminated within an inputimage. FIG. 7B depicts the half-tone image generated in response toinput image FIG. 7A, according to the various embodiments disclosedherein. In this example, the halftone image is placed on a sample 2Kgrid. It should be noted that this half-tone image is substantiallyknown to the display system. FIG. 7C depicts a single pixel PSF on thepremod grid, as blurred by the optic system. FIG. 7D depicts thehalf-tone image of FIG. 7B, as blurred.

FIG. 7E depicts the primary DMD compensation image. It should be notedthat the input image would be represented by the bright centerdot/region—while a halo would form an annular region surrounding thebright center dot/region to compensate for the blurred light field. FIG.7F depicts the resulting screen image after the compensation of FIG. 7Eis applied in order to form the screen image. It should be noticed thatthe screen image may not be radially symmetric, as the system may becorrecting the two errors as previously mentioned herein. Ideally, thescreen image should appear to be the same as the input image, but itexhibits an artifact in FIG. 7F. In this example, the primary DMDcompensation is assumed to be correct and the artifact is a result of animproperly modeled light field caused by an inaccurate PSF model.

PSF Refinement Embodiment

As mentioned, visible artifacts may be captured by camera. When thecaptured image is registered with the premod light field and halftoneimage, it may be used to find a correction to the PSF that may tend toreduce the artifact. The captured image of the artifact may also haveerrors and not be a perfect representation of the real artifact. So thePSF correction may improve the PSF model but may not completelyeliminate the artifact. Using an improved

PSF, the image may be reprocessed and the screen image should exhibit areduced artifact. These steps can be repeated to iteratively reduce theartifact until it is not visible.

FIGS. 8 and 9 show one possible embodiment of a PSF model refinementprocess and/or module. It may be appreciated that such a PSF modelrefinement process/module may be employed one time, multiple times—orpossibly as part of a continual improvement process that may improve theaccuracy of the rendering of desired images during the course ofoperation. As the system starts at 802, an initial PSF model (813) maybe used to display the screen image. Tracking various embodimentsdisclosed herein, input image data 801 may be used to compute a halftoneimage at 804, resulting in halftone image 803. This halftone image maybe used, together with the latest PSF model 813—to compute and/or modelthe light field image 805. In one embodiment, such a light field imagemay be generated by convolving the PSF and the halftone image. Lightfield image 805 may be used to compute primary image (at 808)—to produceprimary image 807. Primary image may be used to display screen image (at810)—to produce screen image 809.

Once the image is rendered as a screen image, a camera picture may betaken at 812 and the screen image may be registered with the premodpixel grid (at 812)—to produce registered screen image 811. Thisregistered screen image may be used for comparison (at 814) to the inputimage. A controller/processor may query whether there are anydifferences between the registered screen image to the input image at816. These differences may be thresholded and/or computed to determineif the difference are greater than a desired amount—e.g., that might bevisible for viewers of the screen image. Alternatively, differences maybe judged by human viewers. If not, then the process may terminate at822—or continue as a corrective process to run continuous process, orrun at a desired time periods.

If there are differences detected at 816, then the system may findand/or compute a PSF correction and improve the PSF model at 820. FIG. 9is one embodiment of a process of computing the PSF correction. At 902,the process/module may divide the registered screen image by the primaryimage. This produces a light field estimate at 805. This light fieldestimate may then be compared with the light field image at 904. Thisproduces a light field error estimate 901. This estimate, together withthe halftone image, may be deconvolved—and/or computed as an inverse—tofind a suitable PSF correction (as 813). This PSF correction may thenform the new PSF model to be used in FIG. 8 .

In general, images derived from the camera picture are estimates becauseof the uncertainty in the camera capture process. The light field may beproduced by the superposition of the PSFs of all the on-pixels in thehalftone image. The deconvolution process attempts to find a PSFcorrection from the estimated light field error by assuming the same PSFfor all on-pixels contributing to the artifact.

Alternative Embodiments from Competing Considerations

As mentioned above, certain assumption and objectives may be used toproduce the new image rendering techniques described herein. Asmentioned, different assumptions and objectives may lead to alternativeembodiments. Such alternative embodiments may be based in DMD contrastratios, PSF sizes and local contrast. For example, the first nonzeropremod level may be achieved by superimposing a field of PSFs to achievea relative flat light field. The relative flat field may be achieved ifthe PSFs are spaced densely enough—as spacing them too far apart mayresult in a field with peaks and valleys that may be large relative tothe desired field level. Given PSF size and shape, the PSF spacingdesired to create a relatively flat field may determine the spacing ofthe halftone pixels that may be turned on to achieve the first nonzeropremod level. This spacing may determine the first nonzero level and thenumber of discrete linear premod levels.

For example, a PSF that is repeated on a 10×10 pixel grid, to achieve arelatively flat field, may require 1 of 100 premod pixels be turned ON.The first nonzero level will be 1/100, the halftone tile size will be10×10, and the number of discrete levels will be 101.

The name first nonzero level indirectly implies that the first level,zero, is 0/100. It may not be, in practice. Unlike all the nonzerolevels, this level may not be determined by the halftone fraction, butby the premod DMD contrast ratio (CR). The level is 1/premodCR and maylikely not be at a level for best performance.

To limit halo size on small bright features, a small PSF andcomplimentary small halftone tile size may be chosen. But the small tilesize limits the number of discrete premod levels. This may not be aconcern at higher levels, but might excessively restrict local contrastat lower levels, particularly levels less than the first nonzero level.Use of the zero level depends on the premod DMD CR; a greater CR may notbe better. Also, depending on the first nonzero level, the zero levelmight be needed to achieve the desired full system CR, achieve black.

Some image features may have spatial frequencies greater than those thatcan be represented by the premod light field. For these image features,the premod light field may be constant, un-modulated. The level of thepremod light field may be determined by the local max of the imagefeature; the level may be a discrete premod level that is greater thanthe local max to avoid bright-clipping. The primary DMD may reduce thepremod light field to produce all levels of the local image feature.Depending on the premod zero level (determined by the premod CR), thefirst nonzero level (determined by the halftone tile size), and theprimary DMD CR, the primary DMD might not have sufficient contrast toproduce the lowest levels, limiting the local contrast of that imagefeature enough to affect its appearance.

Exemplary Images

FIG. 11A is one exemplary image 1102 that depicts a desired source imageto be rendered by the present system and the methods and/or techniquesdescribed herein. As shown, the image is one of church interior that hasdark regions with details on the wall. In addition, there are brightlylit window regions that also exhibit lots of detail therein. FIGS. 11Band 11C depict the upper bound halftone image and the blurred upperbound image, respectively as described herein. FIG. 11D depicts thefinal rendered image as might be produced by one embodiment of thepresent system.

For another exemplary set of figures, FIGS. 12A and 12B show merely onenon-limiting example of some of the processes described herein. In thiscase, a 1-D horizontal line of an example image is used rather thanimage surfaces to more clearly show the quantities described. Luminanceis normalized to 1, the maximum blurred light field level. FIG. 12Ashows the input image line and its bandlimited upper bound and lowerbound. The lower bound is also shown scaled by the contrast ratio of theprimary DMD (primaryCR=1000 here); this is the highest level the lightfield can be such that the primary DMD can still achieve the lowerbound. FIG. 12B shows the light field to be the maximum of the upperbound and scaled lower bound; this value is limited to 1 because that isthe maximum light field level. This light field is at maximum level asmuch as possible given the rules used for this example.

A detailed description of one or more embodiments of the invention, readalong with accompanying figures, that illustrate the principles of theinvention has now been given. It is to be appreciated that the inventionis described in connection with such embodiments, but the invention isnot limited to any embodiment. The scope of the invention is limitedonly by the claims and the invention encompasses numerous alternatives,modifications and equivalents. Numerous specific details have been setforth in this description in order to provide a thorough understandingof the invention. These details are provided for the purpose of exampleand the invention may be practiced according to the claims without someor all of these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

1. A method for calibrating a projector display system, said projectordisplay system comprising a light source, a controller, said controllerreceiving input image data and outputting control signals, a premodmodulator receiving control signals from said controller and light fromsaid light source, a blurring optical system receiving light from saidpremod modulator, and a primary modulator receiving control signals fromsaid controller and light from said blurring optical system, the methodcomprising: displaying the first calibration image by controlling thepremod modulator to display a first dot array and by controlling theprimary modulator to be fully on, such that each pixel of the primarymodulator conveys light received from the premod modulator through theblurring optical system along on optical path that includes a projectionscreen; capturing said first calibration image with an image capturedevice; displaying a second calibration image by controlling the premodmodulator such that each pixel of the premod modulator is at its maximumbrightness level and by controlling the primary modulator to display asecond dot array; capturing said second calibration image with the imagecapture device; using the captures of the first calibration image andthe second calibration image, determining mapping between the premodmodulator and the primary modulator.
 2. The method of claim 1, whereindetermining the mapping between the premod modulator and the primarymodulator comprises: using at least the capture of the first calibrationimage, determining mapping between the premod modulator and the imagecapture device; using at least the capture of the second calibrationimage, determining mapping between the primary modulator and the imagecapture device; and using the determined mapping between the premodmodulator and the image capture device and the determined mappingbetween the primary modulator and the image capture device; determiningmapping between the premod modulator and the primary modulator.
 3. Themethod of claim 1, wherein the premod modulator comprises one of agroup, said group comprising: a DMD, a MEMS, a digital reflector, and ananalog reflector.
 4. The method of claim 1, wherein the premod modulatorcomprises a DMD array that processes a plurality of color channels. 5.The method of claim 1, wherein the primary modulator comprises one of agroup, said group comprising: a DMD, a MEMS, a digital reflector, and ananalog reflector.
 6. The method of claim 1, wherein the primarymodulator comprises a DMD array that processes a plurality of colorchannels.
 7. A method for calibrating a projector display system, saidprojector display system comprising a light source, a controller, saidcontroller receiving input image data and outputting control signals, apremod modulator receiving control signals from said controller andlight from said light source, a blurring optical system receiving lightfrom said premod modulator, and a primary modulator receiving controlsignals from said controller and light from said blurring opticalsystem, the method comprising: conditioning, prior to displaying acalibration image, the projector display system by controlling thepremod modulator such that each pixel of the premod modulator is at itsmaximum brightness level; displaying the calibration image bycontrolling the premod modulator to display a dot array and bycontrolling the primary modulator to be fully on, such that each pixelof the primary modulator conveys light received from the premodmodulator through the blurring optical system along on optical path thatincludes a projection screen; capturing said calibration image with animage capture device; and using the capture of the calibration image,determining mapping between the premod modulator and the primarymodulator.
 8. The method of claim 7, wherein displaying the calibrationimage comprises displaying the calibration image only long enough tofacilitate the capturing of said calibration image with the imagecapture device.
 9. The method of claim 7 further comprising: furtherconditioning, after displaying the calibration image, the projectordisplay system by controlling the premod modulator such that each pixelof the premod modulator is at its maximum brightness level.
 10. Themethod of claim 7, wherein the premod modulator comprises one of agroup, said group comprising: a DMD, a MEMS, a digital reflector, and ananalog reflector.
 11. The method of claim 7, wherein the premodmodulator comprises a DMD array that processes a plurality of colorchannels.
 12. The method of claim 7, wherein the primary modulatorcomprises one of a group, said group comprising: a DMD, a MEMS, adigital reflector, and an analog reflector.
 13. The method of claim 7,wherein the primary modulator comprises a DMD array that processes aplurality of color channels.